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© System and method for improving processing of OCR scanned mail. 



© The invention provides a system and method for 
improving processing of Optical Read Character 
(OCR) scanned mail. Such an automatic mail pro- 
cessing is vested with an interactive learning capa- 



bility. Ambiguous envelope address blocks and in- 
formation fields are resolved using a unique mode of 
human/machine interaction that achieves a sustained 
high rate of automatic mail sortation. 
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This invention relates to the area of automatic 
mail processing and more particularly to an im- 
provement in the use of Optical Character Recogni- 
tion (OCR) in mail processing. 

Worldwide nowadays, several hundred billion 
pieces of mail are delivered by National Postal 
Services. The volume of mail is ever growing at a 
compound rate of approximately 5% a year. 

To handle such massive volume of mail, sev- 
eral methods utilizing automatic means have been 
experimented with and installed on a limited oper- 
ational basis. Quite often, those automatic means 
revolve around OCR. This is the case in the pro- 
cessing system and method disclosed in US Patent 
5,031,223 issued on July 9th, 1991 and assigned to 
International Business Machines Corporation having 
headquarters in the State of New-York, USA, the 
teaching of which will be incorporated hereafter by 
reference. 

A current process of automatic mail sortation 
can be conceptualized into phases: 

Phase 1, image scan and acquisition: it is main- 
ly a hardware pursuit wherein the face of an 
envelope is scanned to 'lift* a replica of what is 
printed into an electronic facsimile that can be 
acted upon by computerized logic. 
Phase 2, analysis of image data: the areas of 
interest are looked for amongst all other in- 
formation on the electronic facsimile of the face 
of the envelope. 

Phase 3, interpretation of image data: it is being 
sought to delineate and interpret information 
fields in the address block corresponding to 
destination inclusive of city, street and recipient. 
Phase 4, mail sortation: the information obtained 
in phase 3 results in the mail pieces being 
- mechanically separated to the desired level to 
facilitate delivery. 
In phase 2 and 3, the mail processing system 
must do sophisticated 'pattern recognition and 
matching' to find the location of address informa- 
tion even when confronted with 'interference' i.e. 
when the areas of interest are lost on the face of 
the envelope amongst a multitude of useless in- 
formation fields. See for example FIG.1 with com- 
mercial and public utility mail showing non stan- 
dard format and extraneous information. Once an 
address location has been determined and the 
character informatinn fiei^Q riicromori o^h ^+;^oMw 
recognized, the constituent address fields must be 
found despite these non standard format or ex- 
traneous information. 

It is unfortunately the current status that OCR 
is prone to misread characters and on occasion 
has difficulty in discerning lines in the address 
block and, when there is interference on the face of 
the envelope, is unable to find the address box. 
When a misread occurs, the mail piece cannot be 



properly sorted and is rejected from the automatic 
process. 

Those mail pieces that fail automatic process- 
ing are sent to off-line LAN terminals/code desks. 
5 Using commercial PS/2 hardware, address image 
is shown to an operator to rekey the address 
information required to effect sortation. Resolved 
mail can then be rerouted to the main line OCR 
sequence. 

10 Very frequently however, codedesk operators 

get 'serial rejects' from a large volume mailer (such 
as a commercial or a public utility one) where 
literally hundreds of envelopes fail for the same 
format related reason: the automatic process is at 
75 that point being drastically affected, where each 
and every following piece of mail in the series of 
rejects is submitted to the operator. The automa- 
tion rate then drops sharply, rending those OCR 
based methods less cost attractive. 
20 'It is therefore an object of the invention to 

provide a system and method for improving pro- 
cessing of OCR scanned mail. 

It is another object of the invention to provide a 
system and method for creating a cooperative real 
25 time link between the operator off-line processes 
and the main line OCR sequence of process for 
better handling serial rejects from the main line 
OCR sequence. 

It is yet a further object of the invention to 
30 provide such a link which allows the main line OCR 
sequence to benefit from the teaching of operator 
inputs regarding a first of a series of rejected mail, 
for resuming automatic processing of following of 
the rejected mail in the series. 
35 These and other objects, features and advan- 

tages of the invention are accomplished by the 
system and method for improving processing of 
OCR scanned mail, disclosed herein. 

Such an automatic mail processing is vested 
40 with an interactive learning capability. Ambiguous 
envelope address blocks and information fields are 
resolved using a unique mode of human/machine 
interaction that achieves a sustained high rate of 
automatic mail sortation. 
45 The address box/line find problems are han- 

dled by prompting the operator to indicate the 
address box/line location which is communicated 
back to the automatic processing routines. 

This information is then used for retrying auto- 
50 matic processing on the next sequentially rejected 
envelope. If the retry succeeds, the same strategy 
is pursued on the succeeding reject and so forth as 
long as the 'recovery chain' holds. This process of 
" cooperative operator automated reject recovery 
55 provides a major increment in automatic processed 
throughput in OCR scanned mail. 

The invention thus includes a method for im- 
proving processing of OCR scanned pieces of mail 
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comprising a main line of automated process and 
off-tines of manual recovery for said pieces of mail 
not being recognized in the main line, including 
applying in off-lines to a current piece of mail, 
teaching from the previously manually recovered 5 
piece of mail, and if retry succeeds, rending the 
current piece of mail to the main line of process, 
and if retry does not succeed, prompting a manual 
recovery of the current piece of mail, memorizing 
teaching from said manual recovery and rending w 
said current piece of mail to the main line of 
process. 

More particularly, the invention includes a 
method for improving processing of OCR scanned 
mail piece having a destination address block in- ;s 
eluding a first routing portion designating at least a 
destination postal location and a second routing 
portion designating at least an addressee, compris- 
ing the steps of capturing an image of said destina- 
tion address block at a sending location, marking 20 
said mail piece with an identity tag, analyzing said 
image to generate a first signal representing said 
destination location, sorting said mail piece at said 
sending location in response to said first signal for 
transport to said destination location, transporting 25 
said mail piece to said destination location, further 
analyzing said image to generate a second signal 
representing said addressee, transmitting said sec- 
ond signal to said destination location, receiving 
said mail piece at said destination postal location, 30 
identifying said mail piece by reading said identity 
tag, and sorting said mail piece in response to said 
second signal for delivery to said addressee, 
wherein said step of analyzing said image to gen- 
erate a first signal includes the steps of automati- 35 
cally locating useful sorting information in the im- 
age using a teaching from the previously pro- 
cessed mail piece, and if automatic location fails, 
presenting said image to an operator and promp- 
ting him to indicate the useful sorting information, 40 
and memorizing said location of said useful sorting 
information as the teaching for the next mail piece 
to be processed. 

These and other objects, features and advan- 
tages of the invention will be more fully appre- 45 
ciated with reference to the accompanying figures: 
F1G.1 is an example of potentially rejected mail 
in an OCR based automatic sorting system. 
FIG.2 is a system diagram of the invention. 
FIG.3 is a functional block diagram of the ar- 50 
chitecture at a sending location, in accordance 
with the invention. 

FIG. 4 illustrates the relationship between the ad- 
dress block on the physical mail piece and the 
captured image of the address block and the 55 
resolved alphanumeric address data. 
FIG. 5 illustrates sample operator assists at a 
workstation, in accordance with the invention. 



FIG.6, consisting of Figs. 6A, 6B and 6C, is a 
process flow diagram illustrating the method of 
the invention as carried out at the sending loca- 
tion. 

For explanation sake, the invention will be de- 
scribed in conjunction with the Deferred processing 
of OCR scanned mail (DOCR) fully explained in 
already mentioned US Patent 5,031,223. But it will 
be obvious to the man skilled in the art that the 
invention can equally be implemented in conjunc- 
tion with other OCR based mail processes. 

The general steps of the invention process are 
as follows: 

1. Mail pieces are input and scanned by a state 
of the art OCR equipment which resolves the 
city/state/zip line of the address. This is suffi- 
cient to route the letter to a district delivery post 
office. Each mail piece is then 'bar coded' with 
an identification number and dispatched to its 
delivery post office. 

2. The image scan of the address block (which 
contains the remaining 3-4 lines of address in- 
formation) is captured, compressed and stored 
on a disk in association with the identification 
number. 

3. While the physical mail pieces are in route to 
their delivery post offices, (i.e., via truck, train, or 
plane, or on a dolly within the same post office) 
their respective address block images are pro- 
cessed off-line in specially configured worksta- 
tions located either at the sending location or at 
a remote location. 

In a workstation, each address block image 
is processed to resolve sortation down to deliv- 
ery sequence as follows: 

a. Off-line OCR is performed on the remain- 
ing 3-4 address lines of the address block in 
either a workstation or in a LAN server pro- 
cessor. 

b. The address data is reviewed against a 
Post Knowledge Base. If there are no appar- 
ent OCR misreads, then the system will: 

- validate and cross check all the ad- 
dress fields including the recipient, 

- resolve any address ambiguities such 
as incomplete address, and 

- derive the delivery sequence within a 
building. 

c. If OCR misreads are encountered, then the 
system will: 

- perform OCR misread correction using 
algorithms for spelling correction. The 
correction candidate information is dis- 
played to the workstation operator 
along with the original image. The oper- 
ator makes the final correction decision. 
This provides advantages over the op- 
erator rekeying the address correction, 
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including economy of keystrokes and 
avoidance of operator errors, 
- make knowledge based disambiguation 
of incomplete address data. 

4. At the completion of Step 3, all the ad- 
dress data required to machine sort the mail 
piece down to delivery sequence will have 
been resolved. The information is then 
batched by destination post office and trans- 
mitted via a high speed telecommunications 
network to the respective destination post 
offices. 

5. The address information is re-associated 
. with each physical mail piece when it arrives 

at the destination post office via the iden- 
tification number previously bar coded on the 
envelope. The physical mail piece and its 
sortation information hence 'come together': 
the sorting is completed down to delivery 
sequence. 

6. At steps 1 and 3, when no address field 
was discerned in the image scan, or when a 
candidate address field was found, but it was 
failed to locate the city, state, zip, street, 
etc. ..fields, then: after the operator on a work- 
station has completed manual address entry, 
he is queried to indicate (for example via 
cursor positioning) the location of the address 
component that the automatic means failed to 
find. This location information is memorized 
and used to coordinate processing of the 
image scan for the envelope having the next 
identification number in ascending order. 

Operational refinements that can be overlaid on 
the process can be found in US Patent 5,031,223. 

Turning now to the figures, FIG. 1 is an exam- 
ple of potentially rejected mail in an OCR based 
automatic sorting system. Indeed the upper en- 
velope shows a lot of extraneous information adja- 
cent to the address field that could prevent auto- 
matic means to discern the latter address field. 

The second envelope shows a frequent case 
where within the address field, useless information 
is being mixed by the mailer with information use- 
ful for sorting, and the automatic means could be 
misled as to where the latter useful information lies. 

FIG.2 is an overall system diagram of the in- 
vention and FIG.3 is an architectural diagram of a 
sending location 10. Mail pieces which originate at 
the sending location 10 are read through an Optical 
Character Recognition machine (OCR) 20. FIG.3 
illustrates a physical mail piece 22 which has a 
destination address block 45 which includes a 
city/state/zip address data field, and an addressee, 
street name and street number data field. The OCR 
20 scans the physical mail piece 22 and captures 
an image 45' of the address block as a two^ 
dimensional array of picture elements in a bit 



plane. The captured image 45' includes an image 
of the city/state/zip information, and further con- 
tains an image of the addressee, street name and 
street number information. The OCR 20 attempts to 
5 resolve the image of the city/state/zip information 
into an alphanumeric character string of resolved 
address data 42, and the system defers the resolu- 
tion of the image of the addressee, street name 
and street number information until later. The case 
io when the image of the city/state/zip cannot be 
automatically resolved will be described further. 

Again on FIG.3 can be seen that a locally 
originated mail piece is input to a conveyor 12 and 
passes beneath the OCR 20 where it is scanned. 
15 The mail piece then continues on the conveyor belt 
and a bar code printer 21 prints a seriaf number 
onto the mail piece 22. In its normal operation, 
OCR 20 will read the second portion of the address 
block 45 consisting of the city, state, country and 
20 zip code destination, and will enter this into the 
resolved address data block 40 in a memory 19. 
As already explained, the case when automatic 
resolution of address block 45 is not possible will 
be described further. 
25 The data processing system includes a CPU 

23 which is connected by means Of a bus 11 to 
the memory 19, the OCR 20 and the bar code 
printer 21. The system further includes worksta- 
tions 31, bar code reader 37, sorting machine 33 
30 connected by connection 35, mass storage 25 and 
communications adapter 27 all interconnected by 
system bus 11. The communications adapter 27 
communicates over communications link 29 to a 
receiving location 28 or an offline or remote. pro- 
35 cessing system as explained in already mentioned 
US Patent 5,031,223. 

The resolved address data block 40 shown in 
FIG.3 has two portions, the first portion 42 stores 
the resolved alphanumeric string for the city, state, 
40 zip code. or country as was recognized by the OCR 
20 in its scanning operation. A second portion 44 of 
the resolved address data block will contain the 
resolved addressee and street name and street 
number information which will -eventually be output 
45 during the course of the operation (see also FIG.4). 

The resolved city, state, zip code and/or coun- 
try information in portion -42 of the resolved ad- 
dress data block 40 is output to the sorting ma- 
chine 33 and is used to physically sort the mail 
50 piece 22 into an . appropriate pocket in the sorting 
machine. The physical pocket in the sorting ma- 
chine 33 is associated with a particular mode of 
transportation, whether by airplane, truck, train or 
other mail transportation medium, which is destined 
55 to the city, state and country named in the destina- 
tion address block 45. 

As the mail piece 22 passes out of the OCR 
20, the bar code printer 21 prints a bar code 
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representing an identification number 24' which will 
allow the mail piece 22 to be re-associated with the 
information in the resolved address data block 40. 
That re-association is made at the receiving loca- 
tion 28 for the mail piece, where the resolved 
addressee, street name and street number informa- 
tion 44 can be associated with a particular mail 
piece 22 by the identity of the identification number 
24'. 

When the mail piece 22 is scanned by the 
OCR 20, the captured image 45* is stored as a two- 
dimensional bit plane of picture elements in the 
mass storage 25, which can be for example a large 
capacity magnetic DASD. The image 45' is stored 
in conjunction with its identification number 24' as 
an image data block 17, and is accessible by its 
identification number. That same identification 
number 24* is also another portion of the resolved 
address data block 40, to facilitate accessing there- 
of. Still further, that identification number 24' is 
imprinted by the bar code printer 21 as the bar 
code onto the face of the mail piece 22. 

If the image of the captured image 45' of the 
address block is successfully resolved, then the 
city, state, zip and country information can be 
output by the OCR 20 in conjunction with the CPU 
23 to the sort machine 33 to physically sort the 
mail piece. 

Prioritization in the operation based upon the 
estimated travel time for locally or remotely des- 
tined mail pieces can be performed according to 
already mentioned US Patent 5,031,223. 

If instead, the image of the captured image 45* 
containing the city, state, zip and/or country in- 
formation is not successfully recognized, then. the 
CPU 23 directs the conveyor 12 to send the mail 
piece 22 to a reject tray 18, mail piece 22 still 
having the bar code 24 imprinted thereon with the 
identification number. Thereafter, by further data 
processing analysis and/or by additional operator 
intervention and interpretation according to proce- 
dures described in US Patent 5,031,223, the un- 
resolved portions of the city/state/zip/country codes 
in the image can be determined and input to the 
portion 42 of the resolved data block 40. 

The present invention introduces a supplemen- 
tary procedure to the above, where the operator is 
queried to indicate, for example via cursor position- 
ing, the location of the address component that the 
system in automatic mode failed to find. Such a 
procedure is described with respect to FIG. 5. 

On a workstation 31, an operator is enabled to 
designate with a mouse 46, portions of the image 
45' that contain useful information 54 for mail sor- 
ting. In another embodiment of the invention, the 
operator is enabled to blot out extraneous informa- 
tion adjacent to useful information. 



Useful information can be the location of the 
destination address block, or location of only part 
of the block (such as the city/state/zip data field, or 
the addressee, street name and street number data 
5 field). 

Such a location information is transferred from 
a workstation 31 to the CPU 23 and memorized. 

Thereafter, all mail, having been held in the 
tray 18 or not, can be fed into the bar code reader 

10 37 associated with the sorting machine 33, for the 
identification number to be read. That number is 
then associated by the CPU 23 with the corre- 
sponding resolved address data block 40 and the 
information in the portion 42 can be accessed to 

15 control the sort machine 33. The sort machine 33 
can then properly sort the mail piece 22 into the 
appropriate transport: after the first sorting opera- 
tion at the sending location 10, the mail piece 22 is 
physically loaded onto a carrier and transported to 

20 the postal destination 28. 

Off-line or remote processing, as already men- 
tioned, allows to resolve the addressee, street 
name and street number information image in each 
captured image 45' of the address block stored on 

25 the mass storage device 25. 

This information, if resolved, will then be en- 
tered as alphanumeric data into the portion 44 of 
the resolved address data block 40. This operation 
is carried out by the CPU 23 using character rec- 

30 ognition algorithms and knowledge base verification 
information. 

If not resolved, and according to the invention, 
the teaching of operator inputs fed back from work- 
station 31 to CPU 23, regarding useful fields in the 
35 preceding rejected mail, are used to try again and 
solve the addressee, street name and street num- 
ber information image in the captured image. 

If this process succeeds, the same strategy is 
pursued on the mail with following identification 
40 numbers, until the 'recovery chain* fails. 

Any of the remote facilities can also benefit 
from the above teaching while in the resolution 
process. 

FIG.6, consisting of FIG.6A, 6B and 6C, is a 
45 sequence of method steps performed at the send- 
ing location 10 to process an incoming mail piece 
in the system shown in FIG. 2. At step 60, a mail 
piece is input to the conveyor 12 and at step 61, 
an identification number is assigned to the mail 
so piece and an address data block is created. Then 
in step 62, the mail piece is scanned by the OCR 
20 and the image 45* is captured and stored in an 
image data block 17 of the memory 19 (step 63). 
Then in step 64, the bar code printer 21 prints the 
55 identification number 24* on the mail piece 22. The 
image 45* of the mail piece is buffered in the 
memory 19 and optionally in the mass store 25 
(step 65). The CPU 23 then in step 66, attempts to 
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locate the address block in the captured image 45'. 
In step 68, if the address block is located, then the 
process passes to step 72. However, if the address 
block has not been located, then at step 69, an- 
other attempt is made by CPU 23, using the teach- 5 
ing from operator inputs regarding useful fields on 
the face of the envelope having the preceding 
identification number. If this new attempt succeeds, 
then the process passes to step 72. However, if it 
fails, in step 70, an operator at one of the work- 10 
stations 31 will assist in locating the address block, 
according to procedures described in US Patent 
5,031,223, then will be queried according to the 
invention, to indicate (for example via cursor posi- 
tioning) the location of the address component that 15 
the automatic means failed to find (see FIG. 5). This 
location information is memorized and will be used 
to coordinate processing of the image scan for the 
envelope having the next identification number in 
ascending order. Then, a branch is made to step 20 
72. 

There, an attempt is made to resolve the image 
portion for the city/state and zip code. In step 74, if 
the city/state/zip code image is resolved, then the 
process passes to step 76 where optional editing of 25 
the image data can be performed and then the 
resolved alphanumeric string for the city/state and 
zip are buffered in portion 42 of the address data 
block 40 and are also output to the mail sorting 
machine 33 (step 78) to sort the mail piece on the 30 
conveyor 12. Branch is then made to step 80. 

However, if the city/state/zip code image is not 
resolved, then the process flows to step 82 where 
the mail piece 22 is stored in the temporary hold- 
ing tray 18. Then then at step 83, another attempt 35 
is made by CPU 23, using the teaching from oper- 
ator inputs regarding useful fields on the face of 
the envelope having the preceding identification 
number. If this, new attempt succeeds, then the 
process passes to step 85. However, if it fails, in 40 
step 84, an operator at one of the workstations 31 
will assist in locating the address block, according 
to procedures described in US Patent 5,031,223, 
then will be queried according to the invention, to 
indicate (for example via cursor positioning) the 45 
location of the address component that the auto- 
matic means failed to find (see FIG.5). This location 
information is memorized and will be used to co- 
ordinate Drocessinn nf tho imon^ +u„ . . 

- C7 — ' « ' • ■ 'uyu O^Cil I I Ul tilt? tip" 

velope having the next identification number in 50 
ascending order. Then, a branch is made to step 
85. 

At step 85, the city/state/zip code information is 
stored in the portion 42 of the address data block 
40. in step 86, the mail piece exits the holding tray 55 
18 and the bar code reader reads the identification 
number for the mail piece and uses the identifica- 
tion number in step 88 as the address for acces- 



sing the city/state/zip information from portion 42 of 
the address data block 40 in the memory 19 and 
this information is then output in step 90 to the mail 
piece sorting machine 33 to sort the mail piece on 
the conveyor belt 12. 

In step 80, the sorted mail piece is transferred 
from the conveyor belt 12 to the transport for 
physical transportation to the destination location 
28. Then the process flows to step 92 where a mail 
piece electronic folder is assembled and handled 
according to US Patent 5,031,223. 

It will be obvious to the man skilled in the art 
that the system as described in the present inven- 
tion implemented at a sending location, can equally 
be applied to the automatic processing that can 
take place at remote/off-line locations. 

Claims 

1. A method for improving processing of OCR 
scanned mail piece (22) having a destination 
address block including a first routing portion 
(42) designating at least a destination postal 
location (28) and a second routing portion (44) 
designating at least an addressee, comprising 
the steps of: 

capturing an image (45') of said destination 
address block at a sending location (10), 
marking said mail piece with an identity tag, 
analyzing said image to generate a first signal 
representing said destination location, 
sorting said mail piece at said sending location 
in response to said first signal for transport to 
said destination location, 

transporting said mail piece to said destination 
location, 

further analyzing said image to generate a 
second signal representing said addressee, 
transmitting (29) said second signal to said 
destination location, 

receiving said mail piece at said destination 
postal location, identifying said mail piece by 
reading said identity tag, and sorting said mail 
piece in response to said second signal for 
delivery to said addressee, 
said method being characterized in that said 
step of analyzing said image to generate a first 
signal includes the steps of: 
automatically locating useful sorting informa- 
tion in the image using a teaching from the . 
previously processed mail piece, and if auto- 
matic location fails, presenting said image to 
an operator and prompting him to indicate use- 
ful sorting information, 

memorizing said location of said useful sorting 
information as the teaching for the next mail 
piece to be processed. 
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2. The method of claim 1, further characterized in 
that the step of prompting the operator in- 
cludes allowing him to point at said useful 
sorting information. 

3. The method of claim 1 , further characterized in 
that the step of prompting the operator in- 
cludes allowing him to blot out useless sorting 
information. 

4. A method for improving processing of OCR 
scanned pieces of mail comprising a main line 
of automated process and off-lines of manual 
recovery for said pieces of mail not being 
recognized in the main line, said method being 
characterized in that it also comprises: 
applying in off-lines to a current piece of mail, 
teaching from the previously manually recov- 
ered piece of mail, and if retry succeeds, ren- 
ding the current piece of mail to the main line 
-of process, and if retry does not succeed, 
prompting a manual recovery of the current 
piece of mail, memorizing teaching from said 
manual recovery and rending said current 
piece of mail to the main line of process. 

5. A system for improving processing of OCR 
scanned mail piece (22) having a destination 
address block including a first routing portion 
(42) designating at least a destination postal 
location (28) and a second routing portion (44) 
designating at least an addressee, comprising: 
means for capturing an image (45') of said 
destination address block at a sending location 
(10), 

means for marking said mail piece with an 
identity tag, 

means for analyzing said image to generate a 
first signal representing said destination loca- 
tion, 

means for sorting said mail piece at said send- 
ing location in response to said first signal for 
transport to said destination location, 
means for transporting said mail piece to said 
destination location, 

means for further analyzing said image to gen- 
erate a second signal representing said ad- 
dressee, 

means for transmitting (29) said second signal 

to said destination location, 

means for receiving said mail piece at said 

destination postal location, identifying said mail 

piece by reading said identity tag, and sorting 

said mail piece in response to said second 

signal for delivery to said addressee, 

said system being characterized in that means 

for* analyzing said image to generate a first 

signal further include: 



means for automatically locating useful sorting 
information in the image using a teaching from 
the previously processed mail piece, and if 
automatic location fails, presenting said image 
5 to an operator and prompting him to indicate 

useful sorting information, 
means for memorizing said location of said 
useful sorting information as the teaching for 
the next mail piece to be processed. 

10 

6. The system of claim 5, further characterized in 
that it includes means for, while prompting the 
operator, allowing him to point at said useful 
sorting information. 

15 

7. The system of claim 5, further characterized in 
that it includes means for, while prompting the 
operator, allowing him to blot out useless sor- 
ting information. 

20 
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